Converting CLC data (108Male/Female Unique Gene Reads Broodstock) into a txt file

In [2]: 
!head /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead.csv

"Feature ID","Range (original values)","IQR (original values)","Difference (original values)","Fold Change (original values)","Expression values","Gene length","Unique gene reads","Total gene reads","RPKM","Means","Expression values","Gene length","Unique gene reads","Total gene reads","RPKM","Means"
"Olurida_trim_nodups_v2reads_contig_1","1,099","1,099","-1,099","-1.102","11,897","1,582","11,897","11,897","132.008","11,897","10,798","1,582","10,798","10,798","90.031","10,798"
"Olurida_trim_nodups_v2reads_contig_10","2,829","2,829","2,829","1.311","9,093","3,992","9,093","9,093","39.984","9,093","11,922","3,992","11,922","11,922","39.393","11,922"
"Olurida_trim_nodups_v2reads_contig_100","1,274","1,274","-1,274","-2.288","2,263","369","2,263","2,263","107.653","2,263","989","369","989","989","35.353","989"
"Olurida_trim_nodups_v2reads_contig_1000","612","612","-612","-1.464","1,930","6,017","1,930","1,930","5.63","1,930","1,318","6,017","1,318","1,318","2.889","1,318"
"Olurida_trim_nodups_v2reads_contig_10000","1,508","1,508","1,508","1.872","1,730","1,472","1,730","1,730","20.63","1,730","3,238","1,472","3,238","3,238","29.015","3,238"
"Olurida_trim_nodups_v2reads_contig_10001","34","34","-34","-1.374","125","904","125","125","2.427","125","91","904","91","91","1.328","91"
"Olurida_trim_nodups_v2reads_contig_10002","57","57","-57","-1.538","163","241","163","163","11.872","163","106","241","106","106","5.802","106"
"Olurida_trim_nodups_v2reads_contig_10003","46","46","-46","-1.167","322","490","322","322","11.535","322","276","490","276","276","7.43","276"
"Olurida_trim_nodups_v2reads_contig_10004","558","558","558","1.552","1,011","2,005","1,011","1,011","8.851","1,011","1,569","2,005","1,569","1,569","10.322","1,569"

Select Columns

In [3]: 
!awk -F ',' '{print $1,$9,$15}' </Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead.csv> /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_2
In [4]: 
!head /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_2

"Feature ID" "Total gene reads" "Total gene reads"
"Olurida_trim_nodups_v2reads_contig_1" "11 "11
"Olurida_trim_nodups_v2reads_contig_10" "9 "9
"Olurida_trim_nodups_v2reads_contig_100" "2 263"
"Olurida_trim_nodups_v2reads_contig_1000" 017" "1
"Olurida_trim_nodups_v2reads_contig_10000" "1 "1
"Olurida_trim_nodups_v2reads_contig_10001" "125" "91"
"Olurida_trim_nodups_v2reads_contig_10002" "163" "106"
"Olurida_trim_nodups_v2reads_contig_10003" "322" "276"
"Olurida_trim_nodups_v2reads_contig_10004" 005" "1

Remove quotations

In [5]: 
!sed 's/"//g' </Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_2> /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_3
In [7]: 
!head /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_3

Feature ID Total gene reads Total gene reads
Olurida_trim_nodups_v2reads_contig_1 11 11
Olurida_trim_nodups_v2reads_contig_10 9 9
Olurida_trim_nodups_v2reads_contig_100 2 263
Olurida_trim_nodups_v2reads_contig_1000 017 1
Olurida_trim_nodups_v2reads_contig_10000 1 1
Olurida_trim_nodups_v2reads_contig_10001 125 91
Olurida_trim_nodups_v2reads_contig_10002 163 106
Olurida_trim_nodups_v2reads_contig_10003 322 276
Olurida_trim_nodups_v2reads_contig_10004 005 1

Rename headings

In [10]: 
!sed 's/Feature ID Total gene reads Total gene reads/contig Female Male/g' </Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_3> /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_4
In [11]: 
!head /Users/srlab/Desktop/108Female_vs_Male_UniqueGeneRead_4

contig Female Male
Olurida_trim_nodups_v2reads_contig_1 11 11
Olurida_trim_nodups_v2reads_contig_10 9 9
Olurida_trim_nodups_v2reads_contig_100 2 263
Olurida_trim_nodups_v2reads_contig_1000 017 1
Olurida_trim_nodups_v2reads_contig_10000 1 1
Olurida_trim_nodups_v2reads_contig_10001 125 91
Olurida_trim_nodups_v2reads_contig_10002 163 106
Olurida_trim_nodups_v2reads_contig_10003 322 276
Olurida_trim_nodups_v2reads_contig_10004 005 1

Converting 106 Male/Female broodstock CLC file to txt file

In [14]: 
!awk -F ',' '{print $1,$9,$15}' </Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads.csv> /Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_2
In [16]: 
!sed 's/"//g' </Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_2> /Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_3
In [18]: 
!sed 's/Feature ID Total gene reads Total gene reads/contig Female Male/g' </Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_3> /Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_4
In [19]: 
!head /Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_4

contig Female Male
Olurida_trim_nodups_v2reads_contig_1 16 16
Olurida_trim_nodups_v2reads_contig_10 9 9
Olurida_trim_nodups_v2reads_contig_100 1 313
Olurida_trim_nodups_v2reads_contig_1000 926 926
Olurida_trim_nodups_v2reads_contig_10000 1 1
Olurida_trim_nodups_v2reads_contig_10001 83 116
Olurida_trim_nodups_v2reads_contig_10002 195 166
Olurida_trim_nodups_v2reads_contig_10003 425 537
Olurida_trim_nodups_v2reads_contig_10004 1 1

Searching for "non-integer"

In [7]: 
awk '$3 ~ /[^0-9]/  print $0'  </Volumes/web/scaphapoda/Hannah/CLC_Data/106Male_vs_Female_bad> 

  File "<ipython-input-7-3f40ad8de9dc>", line 1
    awk '$3 ~ /[^0-9]/  print $0'  </Volumes/web/scaphapoda/Hannah/CLC_Data/106Male_vs_Female_bad>
                                ^
SyntaxError: invalid syntax

Expression Values from CLC 108 Male/Female OlyO

In [12]: 
!awk -F ',' '{print $1,$6,$12}' </Users/srlab/Desktop/106Female_vs_Male_UniqueGeneReads_01.csv> /Users/srlab/Desktop/106Male_vs_Female_02
In [14]: 
!sed 's/"//g' </Users/srlab/Desktop/106Male_vs_Female_02> /Users/srlab/Desktop/106Male_vs_Female_03
In [22]: 
!sed 's/Feature ID Expression values Expression values/contig Female Male/g' </Users/srlab/Desktop/106Male_vs_Female_03> /Users/srlab/Desktop/106Male_vs_Female_04
In [23]: 
!head /Users/srlab/Desktop/106Male_vs_Female_04

contig Female Male
Olurida_trim_nodups_v2reads_contig_1 -1 582
Olurida_trim_nodups_v2reads_contig_10 4 992
Olurida_trim_nodups_v2reads_contig_100 1 313
Olurida_trim_nodups_v2reads_contig_1000 1 926
Olurida_trim_nodups_v2reads_contig_10000 1 472
Olurida_trim_nodups_v2reads_contig_10001 83 116
Olurida_trim_nodups_v2reads_contig_10002 195 166
Olurida_trim_nodups_v2reads_contig_10003 425 537
Olurida_trim_nodups_v2reads_contig_10004 1 005

Expression Values from CLC 108 Male/Female OlyO

In [25]: 
!awk -F ',' '{print $1,$6,$12}' </Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_UniqueGeneRead.csv> /Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_ExpressionValues_1
In [27]: 
!sed 's/"//g' </Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_ExpressionValues_1> /Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_ExpressionValues_2
In [30]: 
!sed 's/Feature ID Expression values Expression values/contig Female Male/g' </Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_ExpressionValues_2> /Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_ExpressionValues.txt
In [31]: 
!head /Volumes/web/scaphapoda/Hannah/CLC_Data/108Female_vs_Male_ExpressionValues.txt

contig Female Male
Olurida_trim_nodups_v2reads_contig_1 -1 582
Olurida_trim_nodups_v2reads_contig_10 2 992
Olurida_trim_nodups_v2reads_contig_100 -1 2
Olurida_trim_nodups_v2reads_contig_1000 1 1
Olurida_trim_nodups_v2reads_contig_10000 1 472
Olurida_trim_nodups_v2reads_contig_10001 125 91
Olurida_trim_nodups_v2reads_contig_10002 163 106
Olurida_trim_nodups_v2reads_contig_10003 322 276
Olurida_trim_nodups_v2reads_contig_10004 1 1

Multiple OlyO Broostock- 106 & 108

In [71]: 
!head /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale.csv

"Feature ID","Range (original values)","IQR (original values)","Difference (original values)","Fold Change (original values)","Expression values","Gene length","Unique gene reads","Total gene reads","RPKM","Expression values","Gene length","Unique gene reads","Total gene reads","RPKM","Means","Expression values","Gene length","Unique gene reads","Total gene reads","RPKM","Expression values","Gene length","Unique gene reads","Total gene reads","RPKM","Means"
"Olurida_trim_nodups_v2reads_contig_1","5593","4494","1468.5","1.116","14553","1582","14553","14553","115.452","10798","1582","10798","10798","90.031","12675.5","16391","1582","16391","16391","214.057","11897","1582","11897","11897","132.008","14144"
"Olurida_trim_nodups_v2reads_contig_10","5833","4954","-3891.5","-1.408","14926","3992","14926","14926","46.926","11922","3992","11922","11922","39.393","13424","9972","3992","9972","9972","51.609","9093","3992","9093","9093","39.984","9532.5"
"Olurida_trim_nodups_v2reads_contig_100","1274","950","629.5","1.543","1328","369","1328","1328","45.168","989","369","989","989","35.353","1158.5","1313","369","1313","1313","73.514","2263","369","2263","2263","107.653","1788"
"Olurida_trim_nodups_v2reads_contig_1000","1360","968","-374","-1.262","2286","6017","2286","2286","4.768","1318","6017","1318","1318","2.889","1802","926","6017","926","926","3.18","1930","6017","1930","1930","5.63","1428"
"Olurida_trim_nodups_v2reads_contig_10000","1879","1508","-1467","-1.95","2785","1472","2785","2785","23.745","3238","1472","3238","3238","29.015","3011.5","1359","1472","1359","1359","19.074","1730","1472","1730","1730","20.63","1544.5"
"Olurida_trim_nodups_v2reads_contig_10001","42","34","0.5","1.005","116","904","116","116","1.61","91","904","91","91","1.328","103.5","83","904","83","83","1.897","125","904","125","125","2.427","104"
"Olurida_trim_nodups_v2reads_contig_10002","89","32","43","1.316","166","241","166","166","8.645","106","241","106","106","5.802","136","195","241","195","195","16.717","163","241","163","163","11.872","179"
"Olurida_trim_nodups_v2reads_contig_10003","261","215","-33","-1.088","537","490","537","537","13.754","276","490","276","276","7.43","406.5","425","490","425","425","17.919","322","490","322","322","11.535","373.5"
"Olurida_trim_nodups_v2reads_contig_10004","1122","1102","-830","-1.813","2133","2005","2133","2133","13.352","1569","2005","1569","1569","10.322","1851","1031","2005","1031","1031","10.624","1011","2005","1011","1011","8.851","1021"
In [74]: 
!awk -F ',' '{print $1,$9,$14,$20,$25}' </Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale.csv> /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_2
In [75]: 
!head /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_2

"Feature ID" "Total gene reads" "Total gene reads" "Total gene reads" "Total gene reads"
"Olurida_trim_nodups_v2reads_contig_1" "14553" "10798" "16391" "11897"
"Olurida_trim_nodups_v2reads_contig_10" "14926" "11922" "9972" "9093"
"Olurida_trim_nodups_v2reads_contig_100" "1328" "989" "1313" "2263"
"Olurida_trim_nodups_v2reads_contig_1000" "2286" "1318" "926" "1930"
"Olurida_trim_nodups_v2reads_contig_10000" "2785" "3238" "1359" "1730"
"Olurida_trim_nodups_v2reads_contig_10001" "116" "91" "83" "125"
"Olurida_trim_nodups_v2reads_contig_10002" "166" "106" "195" "163"
"Olurida_trim_nodups_v2reads_contig_10003" "537" "276" "425" "322"
"Olurida_trim_nodups_v2reads_contig_10004" "2133" "1569" "1031" "1011"
In [76]: 
!sed 's/"//g' </Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_2> /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_3
In [78]: 
!head /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_3

Feature ID Total gene reads Total gene reads Total gene reads Total gene reads
Olurida_trim_nodups_v2reads_contig_1 14553 10798 16391 11897
Olurida_trim_nodups_v2reads_contig_10 14926 11922 9972 9093
Olurida_trim_nodups_v2reads_contig_100 1328 989 1313 2263
Olurida_trim_nodups_v2reads_contig_1000 2286 1318 926 1930
Olurida_trim_nodups_v2reads_contig_10000 2785 3238 1359 1730
Olurida_trim_nodups_v2reads_contig_10001 116 91 83 125
Olurida_trim_nodups_v2reads_contig_10002 166 106 195 163
Olurida_trim_nodups_v2reads_contig_10003 537 276 425 322
Olurida_trim_nodups_v2reads_contig_10004 2133 1569 1031 1011
In [79]: 
!sed 's/Feature ID Total gene reads Total gene reads Total gene reads Total gene reads/contig Female Female Male Male/g' </Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_3> /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_4
In [80]: 
!head /Users/srlab/Desktop/Multiple_OlyO_Broodstock_MalevsFemale_4

contig Female Female Male Male
Olurida_trim_nodups_v2reads_contig_1 14553 10798 16391 11897
Olurida_trim_nodups_v2reads_contig_10 14926 11922 9972 9093
Olurida_trim_nodups_v2reads_contig_100 1328 989 1313 2263
Olurida_trim_nodups_v2reads_contig_1000 2286 1318 926 1930
Olurida_trim_nodups_v2reads_contig_10000 2785 3238 1359 1730
Olurida_trim_nodups_v2reads_contig_10001 116 91 83 125
Olurida_trim_nodups_v2reads_contig_10002 166 106 195 163
Olurida_trim_nodups_v2reads_contig_10003 537 276 425 322
Olurida_trim_nodups_v2reads_contig_10004 2133 1569 1031 1011
In []: